Automatic Speech Recognition (ASR) Feedback for Head Mounted Displays (HMD)

ABSTRACT

Feedback mechanisms to the user of a Head Mounted Display (HMD) are provided. It is important to provide feedback to the user when speech is recognized as soon as possible after the user utters a voice command. The HMD displays and/or audibly renders an ASR acknowledgment in a manner that ensures the user that the HMD has received/understood his voiced command.

RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/905,130, filed on Nov. 15, 2013.

This application is related to the following U.S. patent applications:

U.S. patent application Ser. No. ______, filed Nov. 13, 2014, AttorneyDocket No. 0717.2224-001, entitled “Text Selection Using HMDHead-tracker and Voice-Command.”

U.S. patent application Ser. No. ______, filed Nov. 13, 2014, AttorneyDocket No. 0717.2225-001, entitled “Head-Tracking Based SelectionTechnique For Head Mounted Displays (HMD).”

U.S. patent application Ser. No. ______, filed Nov. 13, 2014, AttorneyDocket No. 0717.2227-001, entitled “Head Tracking Based Gesture ControlTechniques For Head Mounted Displays.”

The entire teachings of the above applications are incorporated hereinby reference.

BACKGROUND OF THE INVENTION

Mobile computing devices, such as a laptop or notebook PC, a smartphone, and tablet computing device, are now common tools used forproducing, analyzing, communicating, and consuming data in both businessand personal life. Consumers continue to embrace a mobile digitallifestyle as the ease of access to digital information increases withhigh speed wireless communications technologies becoming ubiquitous.Popular uses of mobile computing devices include displaying largeamounts of high-resolution computer graphics information and videocontent, often wirelessly streamed to the device. While these devicestypically include a display screen, the preferred visual experience of ahigh resolution, large format display cannot be easily replicated insuch mobile devices because the physical size of such device is limitedto promote mobility. Another drawback of the aforementioned device typesis that the user interface is hands-dependent, typically requiring auser to enter data or make selections using a keyboard (physical orvirtual) or touch-screen display. As a result, consumers are now seekinga hands-free, high quality, portable, color display solution to augmentor replace their hands-dependent mobile devices.

SUMMARY OF THE INVENTION

Recently developed micro-displays can provide large-format,high-resolution color pictures and streaming video in a very small formfactor. One application for such displays can be integrated into awireless headset computer worn on the head of the user with a displaywithin the field of view of the user, similar in format to eyeglasses,audio headset or video eyewear.

A “wireless computing headset” device, also referred to herein as aheadset computer (HSC) or head mounted display (HMD), includes one ormore small, high resolution micro-displays and associated optics tomagnify the image. The high resolution micro-displays can provide supervideo graphics array (SVGA) (800×600) resolution or extended graphicarrays (XGA) (1024×768) resolution, or higher resolutions known in theart.

A wireless computing headset contains one or more wireless computing andcommunication interfaces, enabling data and streaming video capability,and provides greater convenience and mobility through hands dependentdevices.

For more information concerning such devices, see co-pending patentapplications entitled “Mobile Wireless Display Software Platform forControlling Other Systems and Devices,” U.S. application Ser. No.12/348,648 filed Jan. 5, 2009, “Handheld Wireless Display Devices HavingHigh Resolution Display Suitable For Use as a Mobile Internet Device,”PCT International Application No. PCT/US09/38601 filed Mar. 27, 2009,and “Improved Headset Computer,” U.S. Application No. 61/638,419 filedApr. 25, 2012, each of which are incorporated herein by reference intheir entirety.

The present invention relates to use of a Head-Mounted Display (HMD).The HMD is controlled partially by voice-command. The HMD receivesspeech input from the user, and attempts to recognize the receivedspeech, through various processing techniques, as one of a suite ofknown voice commands. It is very important for the HMD to providefeedback to the user when received speech is recognized as a speechcommand, as soon as possible after the spoken command. Doing so helps todeliver a pleasant and efficient experience for the user.

Embodiments provide feedback mechanisms to the user of a HMD, to ensurethat the user feels confident that the unit has understood the spokencommands.

In one aspect, the invention is a method of acknowledging a voicecommand. The method includes receiving, by a headset computer, a voicecommand. The method further includes interpreting the voice command andin response to interpreting the voice command, acknowledging thereceived voice command.

One embodiment further includes converting an utterance, whichcorresponds to the voice command, into an electrical signal. One way ofconverting the utterance is using a microphone, although othertechniques for converting sound into an electrical signal may also beused.

In another embodiment, interpreting the voice command further includesdetermining if the voice command is a member of a predetermined suite ofvoice commands.

In one embodiment, acknowledging the received voice command includespresenting an automatic speech recognition (ASR) notification clue. TheASR notification clue may include a visual clue. The visual clue may bepresented within 500 mS of when the utterance is interpreted, althoughother periods of time between interpretation of the utterance andpresenting the visual clue may be used. The visual clue may beinstantiated on a display within two cycles of the frame rate of thedisplay. In another embodiment, the visual clue remains on the displayfor a predetermined time after the visual clue is instantiated, andremoved from the display after the predetermined time has elapsed.

In some embodiments, the ASR notification clue may include an audioclue, alone or in combination with the visual clue. In one embodimentthe audio clue may be a chirp, although other sounds may alternativelybe used to indicate recognition of an utterance.

In another aspect, the invention is apparatus for acknowledging a voicecommand. The apparatus includes a headset computer configured to receivea voice command, interpret the voice command, and acknowledge thereceived voice command in response to interpreting the voice command.

In one embodiment, the apparatus is further configured to convert, usinga microphone, an utterance corresponding to the voice command into anelectrical signal.

In another embodiment, the headset computer is further configured todetermine if the voice command is a member of a predetermined suite ofvoice commands.

In another embodiment, the headset computer is further configured topresent an automatic speech recognition (ASR) notification clue. The ASRnotification clue may include a visual clue. The visual clue may bepresented within 500 mS of when the utterance is interpreted. The visualclue may be instantiated on a display within two cycles of the framerate of the display. The visual clue may remain on the display for apredetermined time after the visual clue is instantiated, and be removedfrom the display after the predetermined time has elapsed.

In one embodiment, the ASR notification clue includes an audio clue. Theaudio clue may be a chirp.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing will be apparent from the following more particulardescription of example embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingembodiments of the present invention.

FIGS. 1A-1B are schematic illustrations of a headset computercooperating with a host computer (e.g., Smart Phone, laptop, etc.)according to principles of the present invention.

FIG. 2 is a block diagram of flow of data and control in the embodimentof FIGS. 1A-1B.

FIG. 3 is a block diagram of automatic speech recognition (ASR)subsystem in embodiments.

FIG. 4 is a schematic illustration of graphical user interface (GUI)employed in embodiments.

FIG. 5 is a flow diagram according to one embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

A description of example embodiments of the invention follows.

The teachings of all patents, published applications and referencescited herein are incorporated by reference in their entirety.

FIGS. 1A and 1B show an example embodiment of a wireless computingheadset device 100 (also referred to herein as a headset computer (HSC)or head mounted display (HMD)) that incorporates a high-resolution (VGAor better) micro-display element 1010, and other features describedbelow.

HSC 100 can include audio input and/or output devices, including one ormore microphones, input and output speakers, geo-positional sensors(GPS), three to nine axis degrees of freedom orientation sensors,atmospheric sensors, health condition sensors, digital compass, pressuresensors, environmental sensors, energy sensors, acceleration sensors,position, attitude, motion, velocity and/or optical sensors, cameras(visible light, infrared, etc.), multiple wireless radios, auxiliarylighting, rangefinders, or the like and/or an array of sensors embeddedand/or integrated into the headset and/or attached to the device via oneor more peripheral ports 1020 (FIG. 1B).

Typically located within the housing of headset computing device 100 arevarious electronic circuits including, a microcomputer (single ormulticore processors), one or more wired and/or wireless communicationsinterfaces, memory or storage devices, various sensors and a peripheralmount or mount, such as a “hot shoe.”

Example embodiments of the HSC 100 can receive user input throughsensing voice commands, head movements, 110, 111, 112 and hand gestures113, or any combination thereof. A microphone (or microphones)operatively coupled to or integrated into the HSC 100 can be used tocapture speech commands, which are then digitized and processed usingautomatic speech recognition techniques. Gyroscopes, accelerometers, andother micro-electromechanical system sensors can be integrated into theHSC 100 and used to track the user's head movements 110, 111, 112 toprovide user input commands. Cameras or motion tracking sensors can beused to monitor a user's hand gestures 113 for user input commands. Sucha user interface may overcome the disadvantages of hands-dependentformats inherent in other mobile devices.

The HSC 100 can be used in various ways. It can be used as a peripheraldisplay for displaying video signals received and processed by a remotehost computing device 200 (shown in FIG. 1A). The host 200 may be, forexample, a notebook PC, smart phone, tablet device, or other computingdevice having less or greater computational complexity than the wirelesscomputing headset device 100, such as cloud-based network resources. Theheadset computing device 100 and host 200 can wirelessly communicate viaone or more wireless protocols, such as Bluetooth®, Wi-Fi, WiMAX, 4G LTEor other wireless radio link 150. (Bluetooth is a registered trademarkof Bluetooth Sig, Inc. of 5209 Lake Washington Boulevard, Kirkland,Wash. 98033).

In an example embodiment, the host 200 may be further connected to othernetworks, such as through a wireless connection to the Internet or othercloud-based network resources, so that the host 200 can act as awireless relay between the HSC 100 and the network 210. Alternatively,some embodiments of the HSC 100 can establish a wireless connection tothe Internet (or other cloud-based network resources) directly, withoutthe use of a host wireless relay. In such embodiments, components of theHSC 100 and the host 200 may be combined into a single device.

FIG. 1B is a perspective view showing some details of an exampleembodiment of a headset computer 100. The example embodiment HSC 100generally includes, a frame 1000, strap 1002, rear housing 1004, speaker1006, cantilever, or alternatively referred to as an arm or boom 1008with a built in microphone, and a micro-display subassembly 1010.

A head worn frame 1000 and strap 1002 are generally configured so that auser can wear the headset computer device 100 on the user's head. Ahousing 1004 is generally a low profile unit which houses theelectronics, such as the microprocessor, memory or other storage device,along with other associated circuitry. Speakers 1006 provide audiooutput to the user so that the user can hear information. Micro-displaysubassembly 1010 is used to render visual information to the user. It iscoupled to the arm 1008. The arm 1008 generally provides physicalsupport such that the micro-display subassembly is able to be positionedwithin the user's field of view 300 (FIG. 1A), preferably in front ofthe eye of the user or within its peripheral vision preferably slightlybelow or above the eye. Arm 1008 also provides the electrical or opticalconnections between the micro-display subassembly 1010 and the controlcircuitry housed within housing unit 1004.

According to aspects that will be explained in more detail below, theHSC display device 100 allows a user to select a field of view 300within a much larger area defined by a virtual display 400. The user cantypically control the position, extent (e.g., X-Y or 3D range), and/ormagnification of the field of view 300.

While what is shown in FIGS. 1A and 1B is a monocular micro-displaypresenting a single fixed display element supported on the face of theuser with a cantilevered boom, it should be understood that othermechanical configurations for the remote control display device 100 arepossible, such as a binocular display with two separate micro-displays(e.g., one for each eye) or a single micro-display arranged to beviewable by both eyes.

FIG. 2 is a block diagram showing more detail of an embodiment of theHSC or HMD device 100, host 200 and the data that travels between them.The HSC or HMD device 100 receives vocal input from the user via themicrophone, hand movements or body gestures via positional andorientation sensors, the camera or optical sensor(s), and head movementinputs via the head tracking circuitry such as 3 axis to 9 axis degreesof freedom orientational sensing. These are translated by software(processors) in the HSC or HMD device 100 into keyboard and/or mousecommands that are then sent over the Bluetooth or other wirelessinterface 150 to the host 200. The host 200 then interprets thesetranslated commands in accordance with its own operatingsystem/application software to perform various functions. Among thecommands is one to select a field of view 300 within the virtual display400 and return that selected screen data to the HSC or HMD device 100.Thus, it should be understood that a very large format virtual displayarea might be associated with application software or an operatingsystem running on the host 200. However, only a portion of that largevirtual display area 400 within the field of view 300 is returned to andactually displayed by the micro display 1010 of HSC or HMD device 100.

In one embodiment, the HSC 100 may take the form of the device describedin a co-pending US Patent Publication Number 2011/0187640, which ishereby incorporated by reference in its entirety.

The present invention relates to use of an HMD 100 that is capable ofbeing controlled by voice-command. The HMD 100 receives speech inputfrom the user, and attempts to recognize the received speech as one of apredetermined suite of known voice commands.

It is very important for the HMD 100 to provide feedback to the userwhen received speech is recognized as a speech command, as soon aspossible after the spoken command. Doing so helps to deliver a pleasantand efficient experience for the user.

The typical Automatic Speech recognition (ASR) system is set up to waitfor a predetermined period of time once a speech utterance ceases (i.e.,a pause in speech, for example 200 mS) before the system assumes thatthe user has completed the command. So at a bare minimum there willalways be a gap (e.g., 200 mS) after speaking before the ASR subsystemwill provide acknowledgment feedback to the user.

Embodiments of the present invention may provide feedback to the user invisual form, audible form, a command spoken back to the user, or anycombination thereof.

Visual Feedback

One way to notify a user that a command was received, and correctlyinterpreted, is by providing a visual clue, i.e., a clue that can beseen by the user. When a user utters one of a predetermined set of voicecommands, the HMD 100 processes the received sound and evaluates theprocessed information against a set of known voice commands, to generatea proposed interpretation of the sound input.

In accordance with principles of the present invention, the HMD 100 maydisplay, on screen 1010, an alert (for example, a pop-up message orother type of notification) for the user, indicating the voice commandthat the HMD 100 has interpreted. This notification 450 is referredherein to as an ASR (automatic speech recognition) acknowledgement, asshown by the example depicted in FIG. 4. In this example, the user mayhave previously made a selection, perhaps choice “c” from a multiplechoice array of “a, b, c or d.” In order to confirm the choice of “c,”the user enunciates “confirm selection.” Thus, as shown in the exampleof FIG. 4, the HMD 100 presents the “confirm selection” notification 450to the user.

Empirical data suggests that a user ‘responds’ better to a visualnotification presented first with respect to an audio clue. Humans mayperceive a visual, onscreen alert more quickly than a counterpartaudible alert. Based on this assumption, the at least one of thedescribed embodiments seeks to display the ASR acknowledgment 450 assoon as possible.

Example embodiments 100 may instantiate the alert within 500 mS (0.5seconds) of the user voice command being uttered, which may give thesystem 100 crisply responsive feel, with respect to the ASR. Providingsuch a timely prompt, indicating that the utterance was received andcorrectly recognized, may lead to a satisfying experience for the user.

The system 100 seems more responsive if the ASR acknowledgment 450 just‘pops’ up rapidly. A slow fade-in visual effect may add unnecessary timeto the notification and consequently make the system 100 seem lethargic.A fast fade is generally too fast to really be noticed, and so does notprovide any benefit. In an example embodiment, the ASR acknowledgementis completely instantiated within two cycles of the microdisplay's framerate, although in other embodiments the ASR acknowledgement may beinstantiated more quickly or less quickly.

One or more embodiments of the system 100 may leave the ASRacknowledgment 450 displayed/illuminated on the screen (display 1010)for a predetermined period of time to allow an application executing onthe system 100 time to respond. An example amount of time to leave theASR acknowledgement on the screen may be 1 to 3 seconds, althoughshorter or longer times may also be used. After the predetermined amountof time has elapsed, the ASR acknowledgement is removed from thedisplay.

Other embodiments of the system 100 may leave this ASR acknowledgment450 on the screen of display 1010 until the subject application hasfinished its current task, regardless of how long it takes to finish.The disappearance of the ASR acknowledgement 450 may be the visual cluethat indicates to the user that it is permissible to speak (i.e., utter)the next voice command.

Example guidelines for visual design of the foregoing ASR acknowledgment450 may include one or more of: (a) place the ASR acknowledgment in alayer in front of all other content within the screen/display 1010; (b)center on screen both vertically and horizontally; (c) preferably runtext on only one line (i.e., do not wrap text); (d) keep the height ofthe box that contains the ASR acknowledgment at a fixed size; and (e)allow the width to vary, depending on length of ASR command feedback.

Audible Feedback

Another way to notify a user that a command was received and correctlyinterpreted is by providing a clue that can be heard. In one embodiment,an example audio output counterpart to the visual ASR acknowledgement450 described above may be described as an audible “chirp.” A chirp mayconsist of a single, short burst of sound. The chirp may have a constantfrequency, or it may have a frequency that varies as a function of time.

The audio notification may be conveyed to the user by itself, or inconjunction with the visual notification. In an embodiment having acombined visual and audio notification, the audio clue may follow thevisual clue. In this case, once the visual clue 450 is instantiated onthe display screen 1010, the audio (via speaker 9006, FIG. 3) can lag byan additional amount of time (e.g., 500 mS) without a significant effectto the user experience.

The audio chirp may be user configurable, in that the user can choose,for example, to turn the chirp on or off, vary the volume of the chirp,vary the duration of the chirp or modify the frequency characteristicsof the chirp.

Command Spoken Back to User

In alternative audible feedback format (a counterpart to the ASRacknowledgement 450 described herein), the device 100 “speaks back” tothe user (using a text-to-speech facility) the command it hasinterpreted as having been issued (uttered) by the user.

In the illustrated example, the HMD unit 100 confirms to the user thatthe spoken command was correctly received by enunciating (via speaker9006) the command ‘Confirm Selection’ back to the user. The spokencommand acknowledgement is user configurable in that the user can, forexample, choose to turn it on or off, adjust the volume, change thevoice characteristics of the enunciation (e.g., male or female voice,language, etc).

In one embodiment, the user can have either the audible chirp or thetext to speech feedback mechanism, or both, for ASR acknowledgement 450.Although since both are audio clues, embodiments generally do not haveboth working (rendering) at the same time.

The system 100 works by using ‘Text-To-Speech’ software 9035 b toaudibly read the phrase recognized (as one of the phrases that thesystem 100 is ‘listening for’).

In this way, embodiments of the present invention ensure the user feelsconfident that the HMD unit 100 has understood their spoken voicecommands.

In one embodiment the HSC 100 may take the form of the HSC described ina co-pending US Patent Publication Number 2011/0187640 which is herebyincorporated by reference in its entirety.

In another embodiment, the invention relates to the concept of using aHead Mounted Display (HMD) 1010 in conjunction with an external ‘smart’device 200, (such as a smartphone or tablet) to provide information andcontrol to the user hands-free. The invention requires transmission ofsmall amounts of data, providing a more reliable data transfer methodrunning in real-time.

In this sense therefore, the amount of data to be transmitted over theconnection 150 is small—simply instructions on how to lay out a screen,which text to display, and other stylistic information such as drawingarrows, or the background colors, images to include, etc.

Additional data could be streamed over the same 150 or anotherconnection and displayed on screen 1010, such as a video stream ifrequired by the host 200.

FIG. 3 shows an exemplary non-limiting wireless hands-free videocomputing headset 100 under voice command. The user can be presentedwith an image on the micro-display 9010, for example, as output by hostcomputer 200 described above. A user of the HMD 100 can use visualand/or audible ASR acknowledgement software module 9036, either locallyor from a remote host 200, in which the user is presented with visualand/or audible ASR acknowledgement on the microdisplay 9010 and throughthe speaker 9006 of the headset computer 100. Because the headsetcomputer 100 is also equipped with a microphone 9020, the user can uttervoice commands (i.e., make command selections) as illustrated withrespect to one or more embodiments of present invention.

FIG. 3 shows a schematic diagram illustrating the modules of the headsetcomputer 100. FIG. 3 includes a schematic diagram of the operativemodules of the headset computer 100. For the case of ASR acknowledgementin speech driven applications controller 9100 accesses ASRacknowledgement module 9036, which can be located locally to each HMD100 or located remotely at a host 200 (FIGS. 1A-1B). ASR acknowledgementsoftware module 9036 contains instructions to display to a user a screenview 410 of a pertinent message box or the like (examples are detailedin FIG. 4). The graphics converter module 9040 converts the imageinstructions received from the ASR acknowledgement module 9036 via bus9103 and converts the instructions into graphics to display on themonocular display 9010. At the same time text-to-speech module 9035 bconverts instructions received from ASR acknowledgement software module9036 to create sounds representing the contents for the screen view 410to be displayed. The instructions are converted into digital soundsrepresenting the corresponding image contents that the text-to-speechmodule 9035 b feeds to the digital-to-analog converter 9021 b, which inturn feeds speaker 9006 to present the audio to the user. ASRacknowledgement software module 9036 can be stored locally at memory9120 or remotely at a host 200 (FIGS. 1A-1B). The user can speak/utterthe command selection from a screen view 410 and the user's speech 9090is received at microphone 9020. The received speech is then convertedfrom an analog signal into a digital signal at analog-to-digitalconverter 9021 a. Once the speech is converted from an analog to adigital signal speech recognition module 9035 a processes the speechinto recognized speech. The recognized speech is compared against knownspeech, and the ASR acknowledgement module 9036 responds according tothe instructions as described above.

FIG. 5 is a flow diagram according to one of the described embodiments.

It will be apparent that one or more embodiments described herein may beimplemented in many different forms of software and hardware. Softwarecode and/or specialized hardware used to implement embodiments describedherein is not limiting of the embodiments of the invention describedherein. Thus, the operation and behavior of embodiments are describedwithout reference to specific software code and/or specializedhardware—it being understood that one would be able to design softwareand/or hardware to implement the embodiments based on the descriptionherein.

Further, certain embodiments of the example embodiments described hereinmay be implemented as logic that performs one or more functions. Thislogic may be hardware-based, software-based, or a combination ofhardware-based and software-based. Some or all of the logic may bestored on one or more tangible, non-transitory, computer-readablestorage media and may include computer-executable instructions that maybe executed by a controller or processor. The computer-executableinstructions may include instructions that implement one or moreembodiments of the invention. The tangible, non-transitory,computer-readable storage media may be volatile or non-volatile and mayinclude, for example, flash memories, dynamic memories, removable disks,and non-removable disks.

While this invention has been particularly shown and described withreferences to example embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the scope of the inventionencompassed by the appended claims.

What is claimed is:
 1. A method of acknowledging a voice command,comprising: receiving, by a headset computer, a voice command;interpreting the voice command; and acknowledging the received voicecommand, in response to interpreting the voice command.
 2. The method ofclaim 1, further including converting, using a microphone, an utterancecorresponding to the voice command into an electrical signal.
 3. Themethod of claim 1, wherein interpreting the voice command includesdetermining if the voice command is a member of a predetermined suite ofvoice commands.
 4. The method of claim 1, wherein acknowledging thereceived voice command includes presenting an automatic speechrecognition (ASR) notification clue.
 5. The method of claim 5, whereinthe ASR notification clue includes a visual clue.
 6. The method of claim6, wherein the visual clue is presented within 500 mS of when theutterance is interpreted.
 7. The method of claim 6, wherein the visualclue is instantiated on a display within two cycles of the frame rate ofthe display.
 8. The method of claim 6, wherein the visual clue remainson the display for a predetermined time after the visual clue isinstantiated, and removed from the display after the predetermined timehas elapsed.
 9. The method of claim 5, wherein the ASR notification clueincludes an audio clue.
 10. The method of claim 10, wherein the audioclue is a chirp.
 11. An apparatus for acknowledging a voice command,comprising: a headset computer, including a processor, configured to:receive a voice command; interpret the voice command; and acknowledgethe received voice command, in response to interpreting the voicecommand.
 12. The apparatus of claim 1, wherein the headset computer isfurther configured to convert, using a microphone, an utterancecorresponding to the voice command into an electrical signal.
 13. Theapparatus of claim 1, wherein the headset computer is further configuredto determine if the voice command is a member of a predetermined suiteof voice commands.
 14. The apparatus of claim 1, the headset computer isfurther configured to present an automatic speech recognition (ASR)notification clue.
 15. The apparatus of claim 5, wherein the ASRnotification clue includes a visual clue.
 16. The apparatus of claim 6,wherein the visual clue is presented within 500 mS of when the utteranceis interpreted.
 17. The apparatus of claim 6, wherein the visual clue isinstantiated on a display within two cycles of the frame rate of thedisplay.
 18. The apparatus of claim 6, wherein the visual clue remainson the display for a predetermined time after the visual clue isinstantiated, and removed from the display after the predetermined timehas elapsed.
 19. The apparatus of claim 5, wherein the ASR notificationclue includes an audio clue.
 20. The apparatus of claim 10, wherein theaudio clue is a chirp.